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ABSTRACT 

This program evaluation study addressed the struggle of local elementary school speech- 
language pathologists (SLPs) in a school district to provide evidence-based intervention in 
language for students below grade level as required by the U.S. Department of Education. 
Recently, Language Lab™ was published to address the needs of oral language intervention 
prior to special education testing. The district in this study adopted the program. The purpose of 
this program evaluation study, based on the American Evaluation Association evaluation 
standards, was to determine if the Language Lab™ program goals were met in the areas of 
utility, feasibility, and accuracy. The study was grounded in social learning theory. The research 
questions explored SLPs’ perceptions of the components of the program that were effective in 
reducing the need for special education evaluation and improving oral language and narrative 
skills. A sample of 10 SLPs, who implemented Language Lab™ on elementary campuses in the 
district, responded to an electronic survey. Survey responses for rated items were descriptively 
reported and open-ended responses were coded based on program goals and analyzed for 
common themes to answer the research questions. The SLPs reported Language Lab™ as an 
effective program based on improved student oral language skills and a reduction in referrals for 
special education evaluation. Because the program goals also met the evaluation standards, 
recommendations were for Language Lab™’s continued use as a language intervention. Positive 
social change might occur as SLPs in schools use evidence-based interventions such as 
Language Lab™ to improve students’ learning of oral language skills and decrease unnecessary 
placements in special education. 
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INTRODUCTION 

In an age of heightened educational accountability, school districts are expected to provide 
intensive intervention to general education students who struggle academically. Many students 
who struggle in school have difficulty reading, and a significant number of those have weak or 
under developed language learning systems (Paul, 2007). Locally, a school district in southeast 
Texas was faced with the problem of not having evidence-based intervention strategies or 
programs that focused on oral language intervention with general education students who had 
difficulties with oral language skills. An intensive language intervention program was recently 
published for use with students struggling with oral language development. This study was a 
qualitative program evaluation of a newly published intervention program, Language Lab™: 
Response to Intervention for teaching grammar, vocabulary, and storytelling. In the district of 
study, Language Lab™ was implemented by speech-language pathologists (SLPs), with general 
education students who were not meeting grade level expectations in language. The SLPs 
reported the effectiveness of the learning strategies in Language Lab™ as well as the overall 
effectiveness of the program. The findings of this program evaluation were formulated into an 
evaluation report including recommendations and guidelines to assist SLPs in the implementation 
of the program in school districts with children who have weak or under- developed language 
learning systems. 

Definition of the Problem 

When the U.S. Congress authorized the No Child Left Behind (NCLB) law in 2002, it 
included emphasis on improving education through standards-based reform and performance- 
based accountability. The 2004 reauthorization of the Individuals with Disabilities Education 
Act (IDEA) resulted in greater alignment between the two federal education laws. Both NCLB 
and IDEA emphasized the use of scientifically based instructional methods and required 
intensive intervention before referral of a student for special education evaluation to identify a 
possible disability. The local district’s requirement was in accordance with the 2004 National 
Center for Education Statistics and Individuals with Disabilities Education Act (U.S. 

Department of Education [USDE], 2004) which set forth the requirement that school districts 
provide general education intensive intervention prior to referral of students for special 
education testing. 

Teacher accountability for student performance and the requirement to provide 
intensive intervention for general education students struggling to master grade level standards 
resulted in heightened responsibility for teachers. They must have met the needs of struggling 
learners who were potential candidates for placement in special education. Students who 
struggle in reading may also exhibit poor language skills. Therefore, educators often turned to 
SLPs for assistance with language skills in order to improve reading skills. SLPs are viewed as 
experts in language development and have a vital role in providing guidance to educators and 
students regarding oral language skill development as the foundation for reading and writing. 

Locally, the school district struggled to meet the federal mandate to provide intensive 
intervention in the area of oral language prior to referring a student for special education 
evaluation. Nationally, school-based SLPs struggled to meet these same federal mandates. 
Without intensive intervention, students who had difficulty with oral language were often 
referred for a special education evaluation and placed in special education services. Over- 
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identification of language disabilities may contribute to speech impairment being a high- 
incidence disability category under IDEA. 

In response to the need for an evidence-based program that focused on oral and 
narrative language skills, Language Lab™ was recently developed and piloted in the district of 
this study. The program consists of a screener that identifies students who may benefit from 
the program and evidence-based intervention strategies for the teacher or SLP to use to provide 
focused intervention in the language area(s) of need. The program also provides written 
instructions for implementing the program with fidelity and includes directions on collecting 
and effectively analyzing student progress by monitoring data. SLPs who implemented the 
program evaluated Language Lab™ according to the American Evaluation Association 
evaluation standards to determine the utility, feasibility, and accuracy of the program. The 
research questions that guided the study are: 

• How effective were the instructional strategies in the program in reducing the 
need for referral for special education evaluation? 

• In what ways did the program improve oral language and narrative skills? 

• Which components of Language Lab™ were the most effective? 

• In what ways were the printed instructions effective or ineffective in 
implementing components of Language Lab™? 

REVIEW OF THE LITERATURE 

Theoretical Foundation 

An intensive language intervention program is grounded in the social learning theory of 
language development, the descriptive-developmental model, and the zone of proximal 
development (ZPD). Vygotsky, a Russian developmental psychologist, was known for 
emphasizing the importance of social interaction and collaboration as part of the learning 
process (Clabaugh, 2010). Vygotsky (1978) defined the ZPD as a process of learning and 
internalizing learning through varying levels of support by adults. The ZPD focused on the gap 
between a child’s current level of functioning and the child’s potential level of functioning. 
Based on the ZPD, the place where learning occurs was just above the student’s current level of 
functioning (Clabaugh, 2010). Struggling language learners were functioning outside the oral 
and reading proficiency of their age group. Providing tiered instruction for these learners by a 
team of educators supports the precepts of Vygotsky and defines the teaching strategies of RTI 
that are mandated in 2004 by IDEA. Before intervention strategies are recommended, SLPs 
identify the skills needed to develop language for students who exhibit oral language deficits. 
Competent SLPs are trained to identify skills that are needed for students experiencing 
language problems. In addition to identifying skills in language development, the review of 
literature provides information on conditions that explain language development. 

Two conditions that were most effective for the development of specific skills were (a) 
targeting language skills that were absent from the child’s language system, and (b) targeting 
language skills that were used correctly part of the time (Nelson et al., 1996). When a child 
already exhibits correct language skills the majority of the time in their language system, these 
language skills should not be targeted in intervention (Ley, Long, & Linestack, 2003). The 
recommendations by Nelson et al. (1996) and Ley et al. (2003) identified language skills to 
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target in intervention based on Vygotsky’s ZPD. Precisely identifying which language skills 
students can and cannot use and targeting interventions are paramount to eliminating the 
practice of over identifying students for special education. More specifically, Paul (2007) 
recommended that the highest priority of skills to target would be those that the student used 
10% to 50% of occurrence in context. Also a high priority was targeting skills that occurred 
1% to 10% of the time in context but were understood in receptive tasks (Paul, 2007). These 
skills would be considered in the child’s ZPD. Selecting the most appropriate skills to target in 
intervention is critical for efficiently remediating student weaknesses. Targeting the skills in 
the proper sequence requires SLP knowledge of normal language development as well as 
knowledge of the individual student’s language skills. 

SLPs not only apply a broad-based knowledge of social learning theory to identify and 
instruct language deficient students, but also they can apply the descriptive- developmental 
model concept in developing teaching strategies. The descriptive- developmental model of 
language intervention was based on the concept that the normal sequence of language 
development was the guide or standard that was used to determine if a child’s language skills 
were delayed or disordered (Paul, 2007; Wiechmann & Rudebusch, 201 l).The descriptive- 
developmental model compared the child’s language skills to the normal language development 
sequence. The language skills targeted for intervention were targeted in a step-by-step, 
incremental sequence. 

Other language development theorists, such as Bloom and Lahey (1978) and Naremore 
(1997), defined a descriptive-developmental model also known as the communication- 
language approach. In this model, child language was described by various aspects of language 
including form, content, and use. The aspects of language form included skills in phonology, 
syntax, and morphology (Paul, 2007). Language content referred to word meaning and word 
meaning within a sentence. Language use referred to the pragmatics or social use of language 
in a child’s language system (Bloom& Lahey, 1978). A child’s language skills were compared 
to normal language development sequence to help define if interventions were needed and if 
so, what area of language needed intervention. In addition to Language Lab™ utilizing 
learning strategies that fostered language development through the social-learning theory, the 
program used the descriptive-developmental model to determine which language skills 
required intervention. 

Language Lab™, which is a language intervention program that was based on the social 
learning theory, focused on the development of oral language skills based on the premise that 
they are foundational for the development of reading and writing. Most students who struggle 
with reading comprehension also demonstrate weak language comprehension skills, since 
reading comprehension is largely dependent upon language comprehension (Paul, 2007). 
Reading is a language-based skill that enters through a visual portal to access the language¬ 
processing system (Catts & Kamhi, 2005; Wallach, 2005). Since reading is a language-based 
skill, understanding meaning from reading uses the same processes that were used for 
understanding oral language (Paul, 2007). Language Lab™ used various evidence-based 
learning strategies such as scaffolding, pair share activities, and embedding language skills in 
context of connected speech to foster oral language skills, which in turn fostered reading 
comprehension skills. Using the descriptive-developmental language model, the SLP identified 
the current language skills of the student, as compared to normal language development and 
provided evidence- based instructional strategies to foster development of weak language skills 
in the student’s ZPD. 
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By using the descriptive-developmental model to provide focused language 
intervention in RTI, the SLP was able to differentiate between a language disorder and a 
language delay. If the child’s language skills were developing in the sequence of normal 
development but were developing at a slower rate, this was considered a language delay (Paul, 

2007) . If the child’s language skills were not developing in the normal sequence and at a 
slower rate, this may be an indication of a language disorder. Another indication of a language 
disorder is language skills developing in the normal sequence but the student’s language skills 
are not responding as expected in an RTI model. The American Speech-Language-Hearing 
Association (ASHA, 1993) defined a language disorder as impairment in comprehension 
and/or use of spoken, written, and/or other symbol systems. The disorder involved (a) the form 
of language (phonology, morphology, syntax), (b) the content of language (semantics), and/or 
(c) the function of language in communication (pragmatics) in any combination. 

Language difference is another concept considered when analyzing the genesis and 
scope of a language disorder. A language difference may be identified in a child who is 
learning more than one language. Typically, one language will be the primary or first 
language. Often, language aspects or characteristics of the first language (LI) developed in the 
normal sequence but influenced the development of the second language (L2) ( Seitel & 

Garcia, 2009). Language differences were considered to be normal or typical (Seitel & Garcia, 
2009). The SLP must consider the child’s language history, development, and characteristics 
in order to determine if the child has a true language disorder versus a language delay or 
difference. Vygotsky’s and Bloom & Lahey’s theories of learning are compatible with the 
concept of providing focused instruction in an intervention or prevention model. Providing 
intervention services can also be referred to as an RTI model. 

Response to Intervention 

Since the concept of RTI was a relatively new paradigm in public education, data was 
limited on its efficacy. NCLB and IDEA targeted the problem of over-identification of students 
with disabilities, but available instructional strategies were largely ineffective for accelerating 
learning and supporting students with mastery of grade level standards (USDE, 2001; 2004). 
The concept of RTI emerged as a response to these problems. An intervention process was 
required as part of general education in order to support students in general education. IDEA 
included emphasis on the requirement for evidence-based intervention practices through an RTI 
process to address the increasing number of students identified with the high incidence special 
education disability categories of speech impairment and learning disability (National Center 
for Education Statistics, 2004). NCLB raised a critical question among teachers and SLPs as to 
whether all of the students identified with a disability were truly disabled or whether these 
students simply had not been taught with high quality, evidenced-based instruction. 

IDEA defined three elements of evidence-based intervention practices: (a) a requirement 
to use scientifically based instruction, (b) an ongoing evaluation of student responsiveness to 
intervention, and (c) the use of data for the purposes of decision- making (Mesmer & Mesmer, 

2008) . These requirements were the basis for the concept on which RTI was developed. 
Incorporating the premises described in IDEA, Ogonosky (2008) and Rudebusch (2008) defined 
the essential components of an intervention process, also known as RTI, as (a) a belief that all 
children can be taught, (b) an emphasis on early intervention, (c) a multi tier model, (d) the use 
of research-based instruction and intervention, (e) a problem-solving method for making 
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decisions, (f) a method of monitoring student progress, (g) the use of data for decision making, 
and (h) the use of various types of assessment. These eight elements define RTI. 

METHOD 

No Child Left Behind emphasized using evidence-based intensive interventions with 
struggling students prior to referring them for a special education evaluation; however, the 
district of study had no materials to address oral language. Expert SLPs wrote and published a 
new program, Language Lab™: Response to Intervention for teaching grammar, vocabulary, 
and storytelling (Language Lab™). To justify using this new program as a standard teaching 
tool in the district, a program evaluation was needed to determine the overall effectiveness of 
the program. The program evaluation was a qualitative study. The participants in the program 
evaluation were SLPs who had used Language Lab™ at their elementary campus. 

Participation was on a voluntary basis and anonymity was maintained throughout the study. 

SLPs gave voluntary consent for participation in the study through assumed consent by 
completing an anonymous electronic survey. SLPs reported on the improvement of students’ 
language skills, the overall effectiveness of the program, the effectiveness of specific 
components of the program, and the ease of implementation of specific components of 
Language Lab ™ according to the printed instructions. 

Data from an anonymous electronic survey were provided for analysis. The electronic 
survey consisted of open-ended questions and Likert scales which allowed the collection of data 
that answered the research questions. Steps were taken to maximize validity of the study through 
the use of clarifying the bias of the researcher and presenting information in themes developed 
from the data collected. The findings of the program evaluation were compiled and presented to 
the district. The results of the survey were provided to the publisher of the program in the form 
of an evaluation report following the completion of the project study. The evaluation report 
included recommendations for future implementation of the program. 

Qualitative Research Design 

Program evaluations are conducted for a variety of reasons. They may be used to justify 
funding for a program, to measure the effectiveness of a program, for advocacy purposes or to 
fulfill an accreditation requirement. Program evaluations may be qualitative, quantitative, or 
mixed-method in design. This program evaluation was qualitative in nature and designed to 
explore the beliefs and experiences of SLPs regarding the use of Language Lab™ with students 
on their campuses. Qualitative aspects of the utility, feasibility, and accuracy of the program 
were obtained in order to endorse or recommend improvements for using Language Lab™ with 
students who struggle with oral language skills. The expertise-oriented program evaluation was 
selected for four reasons: SLPs (a) understood the components of Language Lab™; (b) knew the 
distinctive, qualifying factors for participants; (c) were qualified to analyze data and (d) could 
evaluate the program as subjective, qualitative experts necessary for program evaluation. This 
program evaluation was aligned with U.S. Congress’ emphasis on the need for scientifically 
based research practices for use with students. A program evaluation of Language Lab™ was 
justified in order to validate, provide recommendations for improvement, or refute the use of the 
program. Therefore, the goal of the Language Lab™ program evaluation was to evaluate the 
overall effectiveness of the program. The American Evaluation Association evaluation 
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standards were used to focus on utility, feasibility, and accuracy of the program. SLPs who had 
implemented Language Lab™ at their elementary campuses responded to the objective and 
subjective items of the survey and provided data to answer the research questions: (a) How 
effective were the instructional strategies in the program in reducing the need for referral for 
special education evaluation, (b) In what ways did the program improve oral language and 
narrative skills, (c) Which components of Language Lab™ were the most effective, and (d) In 
what ways were the printed instructions effective or ineffective in implementing components of 
Language Lab™, considering time sufficiency and clarity? 

This study was designed to make clear connections between the research questions, 
program evaluation standards, survey questions, and the program objectives of Language 
Lab™. The research questions were developed as part of the program evaluation to determine if 
the Language Lab™ program objectives were met. Therefore, the research questions were 
designed to measure the participants perceptions as to whether the program reduced the number 
of referrals for special education evaluation and if the student’s oral language skills improved. 
After these connections were made, the research questions were linked to each of the American 
Evaluation Associations standards of accuracy, feasibility, and utility. The survey questions 
were developed to elicit the perceptions of the participants regarding the validity, value, and 
efficiency of the program. A summary of the alignment of the research questions, Language 
Lab™ program objectives, survey questions, and program evaluation standards is summarized 
in Table 1. See Table 1 (Appendix) 

Participants 

The participants in this program evaluation were SLPs who were employed in the local 
district. The SLPs who had voluntarily implemented Language Lab™ on their elementary 
campus were informed of this program evaluation study. The group of SLPs who implemented 
Language Lab™ was given the opportunity to volunteer as participants in the program 
evaluation with assurance of anonymity. Assumed consent was used by informing the 
participants of their rights and that by completing the electronic survey they were giving their 
consent for participation. The number of SLP participants was determined by those who 
volunteered and completed the electronic survey. It was estimated that there would be 12 
participants; therefore when 10 SLPs responded to the survey, the participation rate was 
considered high. 

Data Collection 

The participants in the program evaluation were instructed via email communication 
on how to access the online survey. Data collection from the survey was used to determine if 
the key objectives of the Language Lab™ program were met. These included the perceived 
effectiveness of (a) the instructional strategies, included in Language Lab™ for improving 
students’ language skills, (b) Language Lab™ as an overall intervention program for 
improving students’ oral language and narrative language skills, (c) specific components of 
Language Lab™, and (d) ease of implementation of specific components of the program 
according to the printed instructions. 

Survey questions were based on the goals, intended outcomes, and instructional 
strategies contained in Language Lab™ as guidelines for the content of the questions. No pre- 
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existing survey was available because of the uniqueness of the program being evaluated. Three 
district SLPs from different districts established validity of the survey. The SLPs critiqued the 
survey questions to ensure that the questions would yield valid and relevant results. The 
electronic survey provided data that was analyzed for themes and inferences of the survey 
results. 

Likert scales were used to rate the efficacy of the instructional strategies in Language 
Lab™, the overall effectiveness of Language Lab™, and the ease of implementation of the 
program components. A list of the specific components of Language Lab™ was provided and 
participants rated the program according to effectiveness. Open-ended questions were used to 
probe for a deeper understanding of the participants’ experience regarding student 
improvement of language skills. All questions included in the survey were designed and linked 
to the program objectives as intermediate indicators of the program evaluation. 

The data obtained were specifically designed to determine the beliefs of SLPs regarding 
the overall effectiveness of Language Lab™ as an evidence-based intervention program for 
students who were struggling with oral language skills. The survey was designed to target 
intermediate indicators of specific components of the program. 

Data obtained on open-ended response questions were entered into qualitative software, 
Nvivo9, for analysis. The use of this software minimized validity threats in the analysis phase of 
the study by removing the potential of researcher bias. 

FINDINGS 

This program evaluation was designed to determine if the program objectives of 
reducing referrals for special education evaluation and improving student oral language skills 
were met. The survey questions were developed to answer the research questions in order to 
determine whether Language Lab™ was an effective, valid, and efficient intervention program 
for use with students who were struggling with oral language skills. The participants’ responses 
to open-ended survey questions and ratings on Likert scales were used to answer the research 
questions which guided the study. 

Research Question 1 

How effective were the instructional strategies in the program in reducing the need for 
referral for special education evaluation? 

In a response to intervention model, students either respond to the intervention provided or they 
do not respond to the intervention provided. Allen, Ukrainetz, and Carswell (2012) used the 
terms “good responders” and “poor responders”. In this study, students were described as 
“responders” or “non-responders”. The SLP participants were asked to classify the students who 
participated in Language Lab™ in one of three categories: (a) exited from intervention, (b) 
continued in intervention, or (c) referred for special education testing. The students who were 
exited from intervention and those who continued in intervention were considered responders to 
intervention. The students referred for special education testing were considered non-responders. 
Overall, the SLPs reported a combined percentage of 70.3% as responders. This was a 
combination of the students who were exited from the intervention process and students who 
continued in Language Lab either to finish the program or due to making progress but needing 
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extended time in intervention. In the non-responder group, 29.7% (n=102) of the students were 
referred for special education testing. See Table 2 (Appendix) 

Therefore, the results indicated a 70.3% (n=242) reduction in referrals for special 
education testing due to the provision of intervention (see Table 2). 

Based on this study, 15 hours of language intervention were sufficient for intervention for the 
students who were exited from the intervention process. The student’s skills improved and they 
were demonstrating grade level oral language skills. Student’s skills were measured through the 
use of pretest and posttest probes which allowed the SLP to note specific language skill 
improvement. For the students who improved their language skills or responded to intervention 
90% (n=9) of the SLP participants reported that Language Lab™ was effective (40%, n=4) or 
very effective (50%, n=5) as an evidence-based intervention program. See Table 3 (Appendix) 
Regarding the overall effectiveness of Language Lab™ for students who needed to 
continue in Language Lab™, 60% (n=6) of the SLP participants rated the overall effectiveness of 
the program as effective (40%, n=4) and very effective (20%, n=2). 

Additionally it was noted that 40% of the SLP participants rated the overall 
effectiveness of the program as neutral (see Table 3). A possible explanation of these results 
was that the study did not require the SLPs to report on students who had completed the full 
Language Lab™ program. These results included students who were responding at the 
expected rate but needed to complete the program as well as students who were responding to 
the interventions but needed more than 15 hours of language intervention. 

While the Language Lab™ intervention program was effective for identifying 
responders and non-responders, the SLP participants reported 29.7% (n=102) of the students 
who participated in the program were non-responders. These students were referred for 
special education testing for a possible speech-language impairment and/or other special 
education disability. Through the provision of intervention in Language Lab™, the SLPs were 
able to identify students who did not respond to intervention to determine whether they needed 
specialized instruction for a special education disability. Even though 29.7% (n=102) of the 
students were referred for special education testing, 70% (n=7) of the SLPs reported that 
Language Lab™ was an effective (40%, n=4) or very effective (30%, n=3) intervention 
program to use in an RTI model (see Table 3). 

Even though the non-responder students (29.7%, n=102) were referred for a special 
education evaluation, it was important to know if the SLPs believed the data obtained during 
intervention was valuable as part of the evaluation data. Ninety percent (90%, n=9) of the 
SLPs reported that the data obtained in Language Lab™ as valuable data (40%, n=4) or very 
valuable (50%, n=5) data to use as part of the evaluation. See Table 4 (Appendix) 

The data obtained on the student’s pretest, posttest, and specific examples of student 
productions during the sessions helped identify strengths and weaknesses in the student’s 
language systems. The data obtained regarding the student’s response to language 
intervention allowed the SLP to compare the child’s language skills to a normal development 
sequence to determine whether the student was demonstrating a language delay, language 
difference, or a language disorder. 

In the open ended questions, one SLP made the comment that “selecting the right kids 
for intervention is key”. She continued by explaining that the students, who performed very 
poorly on the pretest, may need to be referred for testing rather than participate in intervention. 
This feedback seemed to align with Paul (2007) who recommended targeting students who 
were correctly using the skill 10% to 50% of the time. These findings indicated a need for 


A program evaluation, page 9 


Journal of Instructional Pedagogies 


Volume 15 - October, 2014 


future research regarding the selection of students who were most appropriate for a response to 
intervention program. 

The survey questions were designed to address the program evaluation standards of 
utility, feasibility, and accuracy. Again, utility referred to the value of the program, feasibility 
referred to the efficiency of the program, and accuracy referred to validity of the program. An 
example of survey questions that addressed the utility of the program were questions that asked 
the SLPs about the value of the intervention data when a student was referred for a special 
education evaluation. The feasibility of the program was measured by the SLP reports of 
student progress and responsiveness to the intervention. Feasibility of the program was also 
measured through survey questions that asked the SLPs to rate the overall effectiveness of the 
program. The accuracy or validity of the program was measured by asking the SLPs about 
student outcomes. When a student’s posttest probe scores improved it demonstrated the 
instructional components were valid in changing the students’ oral language skills. Based on 
SLP responses, this program evaluation study supported Language Lab™ as an evidence-based 
intervention program for oral language skills. This was evidenced by the effectiveness ratings 
(see Table 3), reported value of the intervention data (see Table 4), and ratings of student 
responsiveness to the program (see Table 2). Based on these findings it was determined that 
Language Lab™ positively met the program evaluation standards of utility, feasibility, and 
accuracy. 

Research Question 2 

In what ways did the program improve oral language and narrative skills? 

Through the use of visual inspection and qualitative software, Nvivo9, themes emerged 
from the SLPs responses. The SLPs were asked how the student’s oral language (syntax) skills 
and narrative skills improved through participation in Language Lab™. 

The information reported by the participants was used to meet the program evaluation 
standard of utility or value of the program. Common term searches revealed two common themes 
that emerged from the responses regarding the improvement in the area of syntax and 
generalization of the new skills to other settings or language contexts. Each respondent (90%, 
n=9) made a comment related to some form or type of improvement, except one (10%, n=l) 
respondent who did not answer this question. Various types of responses related to improvement 
included comments such as “syntax improved notably”, “syntax improved by direct instruction”, 
“some improved, especially with irregular past tense and plurals”, and the “most gain was noted 
in the students’ use of compound sentences, pronouns, prepositions, irregular past tense, and 
noun-verb agreement” (see Table 5). The second noted theme that emerged, which went hand in 
hand with the comments regarding improved skills, were references to generalization or carryover 
of the improved language skills. The participants made comments such as “students began self- 
correcting”, “techniques were able to be earned over in the hallways.. .and at home”“, students 
were able to generalize”, “all of the students transferred the new skills from correct use in drill to 
correct use in connected language”, and “objectives were evident in their conversational speech.” 
See Table 5 (Appendix) 

In regard to improvement in narrative skills, 90% (n=9) of the SLPs reported 
improvement through comments like “narrative skills improved.. .they could tell a story”, “skills 
improved because they were taught the parts of the story”, “students were able to elaborate more 
details”, “students improved greatly”, “students included more story grammar elements”, and 
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“students engaged in improving their literacy skills”. Also supporting improvement of skills, the 
responses from the SLPs referenced the carryover or generalization of skills. Comments related to 
generalization included statements like “teachers reported that there was carryover to the 
classroom”, “they were able to answer in complete sentences”, “students transferred this skill to 
show proficiency in narratives.” See Table 6 (Appendix) 

An additional theme was evident in the responses related to narrative skills. This dealt 
with references to teaching strategies that facilitated improvement in skills. The strategies 
mentioned included the use of visual prompts/supports, scaffolding, corrective feedback, and 
naturalistic modeling (see Table 6). 

Based on the specificity and prevalence of comments from nearly every SLP participant 
regarding the various ways students improved their syntax or oral narrative language skills, the 
findings of this study supported the value or utility of Language Lab™. The void of negative 
comments or negative examples of ways students language skills changed also supported the 
findings that validated Language Lab™. 

Research Question 3 

Which components of Language Lab™ were the most effective? 

Since this study was a qualitative program evaluation, the SLPs were asked to rate the 
effectiveness of each of the components of Language Lab™ relative to working on the specific 
target skills. The responses were reported using a Likert scale in order to determine which 
components of the program were rated as effective or very effective. 

The responses served to meet the program evaluation standard of accuracy. The 
Language Lab™ Story Station was rated the most effective component of the program. Story 
Station was rated as very effective by 80% (n=8) and effective by 20% (n=2) of the SLPs. The 
second highest overall effectiveness ratings by the SLPs were for Skill Drill. Sixty percent 
(n=6) of the SLPs rated Skill Drill with an overall effectiveness rating of very effective and 
30% (n=3) as effective. The other three instructional components received a combined 
effectiveness rating of effective and very effective to total 70% (n=7) (see Table 7). Based on 
SLP ratings for each of the instructional components of Language Lab™, conclusions were 
drawn that the instructional components validly addressed the students’ oral language skill 
deficits. See Table 7 (Appendix) 

Research Question 4 

In what ways were the printed instructions effective or ineffective in implementing 
components of Language Lab™, considering time sufficiency and clarity? 

As part of a program evaluation, it is important to include an evaluation of the 
mechanics of the program including the clarity of the printed instructions in the manual and 
the time allotted for each instructional component. By combining the SLP feedback on time 
sufficiency and instruction clarity, conclusions were drawn regarding the overall effectiveness 
and feasibility of the program in this regard. Therefore the program evaluation feasibility 
standard was addressed through this series of questions. In the printed instructions the amount 
of time needed for each station or component was provided for the reader. According to the 
survey results, the majority of the SLP participants rated the printed instructions either 
effective or very effective for each instructional component. Specifically, 80% (n=8) of the 
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SLPs rated the instructions for Skill Drill and Curriculum Connections as effective or very 
effective. For Talk Aloud and Listen ‘n Learn printed instructions, 70% (n=7) of the SLP 
participants rated the clarity of the instructions as effective or very effective. The highest 
percentage of SLP participants (90%, n=9) rated the clarity of the instructions for Story 
Station as effective and very effective. See Table 8 (Appendix) 

In regard to the sufficiency of the time allotted for each instructional component, the 
Listen ‘n Learn station was rated the highest of all of the learning components. Ninety percent 
(90%, n=9) of the SLPs reported the time allotted for Listen ‘n Learn as sufficient or very 
sufficient. The remaining participant (10%; n=l) reported the time allotted for Listen ‘n Learn 
as neutral. The time allotted for Curriculum Connections was rated as sufficient or very 
sufficient by 80% (n=8) of the participants. The remaining two participants rated the time 
allotted for Curriculum Connections as neutral and somewhat sufficient. The Talk Aloud 
station was rated sufficient and very sufficient by 70% (n=7) of the participants for the time 
allotment. Two participants rated that time allotted for Talk Aloud as neutral and one (10%) 
rated it as somewhat sufficient. Only nine participants responded to the question regarding the 
time allotted for Skill Drill. A sufficient or very sufficient rating was given by 66.6% (n=6) 
SLPs for Skill Drill. While the Story Station was rated the lowest regarding time allotment, 
50% (n=5) of the SLP participants rated the time allotted for as sufficient or very sufficient. 

This resulted in 30% (n=3) of the SLPs rating the time allotted for Story Station as not 
sufficient. Two participants (20%) rated Story Station as neutral for the time allotted. Since 
additional comments or explanations were not elicited for these questions, it was interpreted 
that those who stated the time allotment was insufficient meant that there was not enough 
time allotted for this learning station. See Table 9 (Appendix) 

There were several possible variables that may have influenced or affected the time 
factor. These variables included the number of target skills the students were addressing, how 
low the students scored on the Language Lab™ screener, and the pacing of the SLP as she 
conducted the lesson in Story Station. For example, if a student did not get any items on the 
screener correct, they would have a higher number of target skills to address; therefore, it 
would be expected to require more time. Since this was an RTI program, the lessons were not 
intended to be slowly paced as one would do in a therapeutic session but should be presented 
at a normal pace in order to measure how the students responded to intervention. 

CONCLUSIONS 

Language Lab™ is an evidence-based intensive intervention program designed for use 
with students who are struggling to meet grade level expectations in oral and narrative 
language skills. In this qualitative study, an expertise-oriented program evaluation was 
conducted to evaluate Language Lab™ based on the American Evaluation Association 
evaluation standards that measure the utility, feasibility, and accuracy of the program. An 
anonymous electronic survey was conducted to collect data on the program. The data 
obtained through the survey were analyzed and the research questions were answered with 
thick descriptions of the SLPs experiences with the program. 

Based on the findings of this program evaluation, the data gathered supported Language 
Lab™ as an evidence-based intervention program for oral language skills. The Language 
Lab™ program objectives of improving students’ oral language skills and reducing the need for 
referral for special education evaluation through the use of evidence-based instructional 


A program evaluation, page 12 


Journal of Instructional Pedagogies 


Volume 15 - October, 2014 


components were met. The goal of an RTI model was to determine if students responded to the 
intervention that was provided. The second objective of Language Lab™ was to reduce the 
referrals for special education evaluation. In this study 70.3% (n=242) of the students were 
classified as responders to the intervention provided through the use of Language Lab™. 
During this study, 38.1% (n=131) of students were exited from intervention and did not need a 
referral for a special education evaluation. The remaining 32.3% (n=l 11) students in the 
responder category continued in intervention therefore the final outcome regarding the need for 
a special education referral remained to be seen. Based on these two responder groups of 
students, the results indicated a 70.3% (n=242) reduction in referrals for special education 
evaluation as a result of intervention received in Language Lab™. 

In this study, the SLPs consistently reported that the student’s syntax and narrative 
skills improved. Therefore, this program evaluation supported Language Lab™ as an effective 
evidence-based intervention program for oral language skills. From a program evaluation 
standard perspective, the findings of this study provided data that validated Language Lab™ in 
all three areas of the American Evaluation Association program evaluation standards of utility, 
feasibility, and accuracy. 

This program evaluation affected social change by substantiating an evidenced- based 
program that served as an early intervention program for students who were deficient in oral 
language skills. Students who responded to the instructional strategies presented in Language 
Lab™ avoided erroneous placement in special education. Not only were the students in the 
district of study provided the intervention for improvement of oral language skills and avoided 
special education placement, but also SLPs in a broader educational setting will benefit from 
using Language Lab™ as an evidence-based intervention program. An overarching social 
change will occur as districts comply with the requirements of NCLB and IDEA through the 
provision of intensive intervention to students who struggle to meet grade level expectations. 
Districts that implement Language Lab™ will acquire the means to leave no child behind 
specifically in the area of oral language skills. 
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APPENDIX 


Table 1 


Connection Between Research Questions, Program Objectives, Survey Questions, and 
Program Evaluation Standards 

Evaluation 


Research Questions 


Program Objectives Survey Questions Standards 


1. How effective were 
the instructional strategies 
in the program in reducing 
the need for referral for 
special education 
evaluation? 


Reduce the number of 
referrals for special 
education evaluation 


Questions 1-7 


Feasibility 

Accuracy 

Utility 


2. In what ways did the 
program improve oral 
language and narrative skills? 


Improve oral narrative 
language skills 


Questions 8-9 Utility 


3. Which components of Both objectives 

Language Lab™ were most 

effective? 


Questions 10-14 Accuracy 


4. In what ways were the Both objectives 

printed instructions effective 

or ineffective in 

implementing components of 

Language Lab™, 

considering time sufficiency 

and clarity? 


Questions 15-24 Feasibility 
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Table 2 

Student Outcomes of Language Lab™ Intervention 



Responders 


N on-Resnonders 


Exited 

Continued 

Referred 

SLP Report 

77 % 

77 

% 

77 % 

Students 

131 38.1 

111 

32.2 

102 29.7 


Table 3 


Overall Effectiveness Ratings for Students 


Not Somewhat Very 

Group Rating Effective Effective Neutral Effective Effective 
n % % n % n % n % 


Exited students 

- 

1 

10% 

4 

40% 

5 

50% 

Continued students - 

- 

4 

40% 

4 

40% 

2 

20% 

Referred students 

2 

20% 1 

10% 

4 

40% 

3 

30% 


Table 4 

Value Ratings for Students Intervention Data 


Rating 

Not 

Effective 

77 % 

Somewhat 
Effective 
n % 

Neutral 
n % 

Effective 

77 % 

Very 

Effective 

77 % 

SLP rating for 
value of data 

_ 

_ 

1 10% 

4 40% 

5 50% 
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Table 5 


Participant Comments Regarding Student Syntax 


Respondent 

Theme 

Improvement 

Theme 

Generalization 

SLP1 

Syntax improved notably 

Student’s [sic] began self- 
correcting 

Techniques.. .carried over 

SLP2 

Syntax improved 

in the hallway.. .and at home 

SLP3 

Syntax has improved 

Students were able to 
generalize 


Theme 

Strategies 


SLP4 

SLP5 

SLP6 

SLP7 


SLP9 

SLP10 


Some improved, especially 
with irregular past tense 
Some continued to struggle 

Students were able to self- 
correct 

All of the students transferred 

1 ___ 

Most gain was noted Use in connected language 

Students began to make Natural 

connections modeling 

Corrective 

Feedback 

Scaffolding 

Students improved 

Evident in their conversational 
speech 
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Table 6 


Participant Comments Regarding Student Narrative Skills 



Theme 

Theme 

Theme 

Respondent 

Improvement 

Generalization 

Strategies 

SLP1 

Narrative skills improved 


Visual prompts 

SLP2 

Narrative language skills 
improved 


Visual prompts 

SLP3 

Elaborated more details 



SLP4 

Students improved greatly 

Teachers reported 
carryover to 
classroom 


SLP5 

Able to answer questions 
without verbal prompts 

Able to answer in complete 




sentences 




Comprehension of the 
material greatly improved 
Could retain the information 



SLP6 

Develop their understanding 
of making predictions 

All students used the skills 
Showed proficiency 

Included more story grammar 
elements 

Transferred skills 


SLP7 

Immediate ability to recall 

Show progress on 



Better understanding 

their own 



More confident with 

Language link to 



identifying story components 

literacy 


SLP8 

Story telling skills improved 
Stories sequenced correctly 
Inclusion of story elements 



SLP9 

Able to answer specific 
questions 



SLP10 

Somewhat effective 
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Table 7 


Overall Effectiveness Ratings for Instructional Components 


Instructional 

Component 

Not 

Effective 
n % 

Somewhat 
Effective 
n % 

Neutral 

n % 

Effective 
n % 

Very 
Effective 
n % 

Skill Drill 

_ 

1 

10 

- 


3 

30 

6 60 

Talk Aloud 

1 10 

- 


2 

20 

1 

10 

6 60 

Listen ‘n Learn 

- 

1 

10 

2 

20 

4 

40 

3 30 

Story Station 

- 

- 


- 


2 

20 

8 80 

Curriculum 

Connection 


1 

in 

9 

90 

9 

90 

s so 




Table 8 






Effectiveness Ratings of Clarity of Printed Instructions 





Instructional 

Component 

Not 

Effective 
n % 

Somewhat 
Effective 
n % 

Neutral 

n % 

Effective 
n % 

Very 
Effective 
n % 

Skill Drill 

_ 

2 

20 

_ 


4 

40 

4 40 

Talk Aloud 

1 10 

1 

10 

1 

10 

3 

30 

4 40 

Listen ‘n Learn 

1 10 

1 

10 

1 

10 

4 

40 

3 30 

Story Station 

- 

1 

10 

- 


4 

40 

5 50 

Curriculum 

Connection 


_]_ 

_ 

_1_ 

_1X1_ 

_ 

-Ml _ 

_4_ ALL 
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Table 9 


Time Sufficiency Ratings of Clarity for Instructional Components 


Instructional 

Component 

Not 

Sufficient 

n % 

Somewhat 

Sufficient 

n % 

Neutral 
n % 

Sufficient 

n % 

Very 

Sufficient 

n % 

Skill Drill 

1.1 11.1 

- 

2.2 22.2 

3.3 

33.3 

3.3 

33.3 

Talk Aloud 

- 

1 

10 

2 

20 

4 

40 

3 

30 

Listen ‘n Learn 

- 

- 

1 

10 

6 

60 

3 

30 

Story Station 

3 30 

- 

2 

20 

4 

40 

1 

10 

Curriculum 

Connection 


1 

in 

1 

in 

s 

sn 

'X 

^n 




